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BACKGROUND OF MASS DATA STORAGE SYSTEMS 


NDADS: 

Prototype of the Hubble Space Telescope Data Archive and Distribution 
Service (HST-DADS) contracted to Loral AeroSys in 1989. Evolved as the 
Astrophysics and Space Physics archiving system for the National Space 
Science Data Center to maintain a mix of near and on-line data and manage a 
deeper data storage archive 

GDAAC/V.O: 

EOS prototype archive and distribution systems initiated in FY91 and planned 
for operational availability in FY94. One of nine geographically distributed 
discipline-oriented interoperable DAAC's 

M(DS)2 : 

A mass storage and delivery system serving more than 1400 users within the 
NASA Computational Science Center at Goddard that has to manage both the 
high-speed computer-generated simulation data, as well as space-borne 
observational data 


SYSTEM REQUIREMENTS 



NDADS 

GDAAC V.O 

M(DS)2 

NEAR-ONLINE 
STORAGE /DEEP 

2.6 TB/6 TB 
16GB DASD 

10TB/3TB 

16GB/DASD 

7TB/35TB 
240 GB/DASD 

SCALABLE UP TO 

10 TB/50TB 
100 GB DASD 

18 TB 

100 GB/DASD 

225 TB/500 TB 
3 TB/DASD 

INGEST 

(RATE) 

13 GB/DAY 

30 GB/DAY 

90 GB/DAY 

DISTRIBUTE 

(RATE) 

1050 MB/DAY -NET 
700 MB/DAY -TAPE 
100 PHOTOS/DAY 

150 GB/DAY 

100 GB/DAY 

PEAK 

CONCURRENT 

USERS 

146 

240 CATALOG QUERIES/HR 

100 

MIN. 128 

MIN. 32 SIMULTANEOUS 
FTP TRANSFERS 
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SYSTEM FUNCTIONAL CHARACTERISTICS 


Data anil Metadata Functions 


Network Access (Ethernet, FDD1, DecNet, UltraNet) 


Security 


Integrity and Quality Control 


Automated Data Migration and Compaction 


User Ingest and Retrieval 


Remote Ordering and Delivery Service 


Catalogues and Inventories 


Browse - On-Line 


Interoperability 


Database Queries and Subsetting 


Portable Software Operating Systems 


Incorruptible Archive 


Remote Back-Up/Safe Store 


Data Compression 


Redundancy (NSPOF) 


Scalable Upgrade 


Accounting & Monitoring 


NDADS I GDAAC MDSDS 



NDADS Data Sources: 


Astrophysics: 


ect 


EXOSAT 


luiPiEnnaiH 


HEAO-3 


HEASARC 


EINSTEIN 


ASTROD 


XTE 


CRO 


ROSAT 


VELA 5B 


IUE 


EUVE 


HST 


COBE 


IRAS 


VLA 


Miscellaneous I ADCCaUl 


TOTAL 



10 


2 


200 


20 


launch 1993 


launch 1996 


100 




34K 


10K 


100K 


15K 







Space Physics 


Mission 


SKYLAB 


i>j u fci. ' i - ' ■ n i 


P8 


-GEOTAIL 


IHil 


DE-1 


VOYAGER 


TOTAL 


Granules I Archived | Total Size 


10 GB 


.4 GB 


.4 GB 


1.0 GB 


1GB 


100 GB 


2 GB 


114 GB 
































































































































Gigabytes 



Science Project Data Products 


Project 


(JARS 


SeaWiFS 


Atlas/ 

ATMOS 


TOMS2 


AVHRR 

Pathfinder 


TOVS 

Pathfinder 


Product Description 



• Ocean pigment, chlorophyll a concentrations 

* 5 water leaving radiances, 3 aerosol radiances 
» Pifftue attenuation coefficient 


• Profiles of 3t+ upper atmosphere trace species 

• Upper atmospheric temperature profiles 


• Total oione, effective tropospheric reflectivity 
< hachscattered UV radiances (313 • 340 nm) 


• Binned 5 channel clear sky radiances 

• Daily cloud fraction, height, and reflectivity at 9 km and I degree spatial 
resolutions 

• D*Hy, weekly and seasonal surface reflectance NDVI at 9 km resolution 

• Daily Surface albedo at 9 km resolution 

« Aerosol optical thickness, longwave surface flu* 


• Profiles of atmospheric temperature, humidity, and geopotenlial height 

• Precipitable water in 0 tropospheric layers, total ozone, and Iropopause 
pressure 

• Surface air and skin temperatures, 3.7 micron bidirectional surface 
reflectance, and 50 GHz suirface microwave emissivily 

• Cloud fraction, cloud top pressure, precipitation estimate, visible reflectance, 
outgoing longwave radiation, and longwave cloud forcing 


Goddard Version 0 DA AC 
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M(DS)2 data sources 


PROJECTS: NIMBUS/TOMS 

ISTP 
IUE 
GRO 


MODELING: DATA ASSIMILATION 

COUPLED OCEAN/ATMOSPHERE/STRATOSPHERE 
GEODYNAMICS 

SPACE PHYSICS PLASMA MODELING 


ANALYSIS: TOVS PATHFINDER 

ALGORITHM DEVELOPMENT (TRMM, MODIS...) 
HST IMAGE DEBLURRING 


HPCC: EARTH AND SPACE SCIENCE TESTBEDS 


Mass Storage Hardware Systems 


NDADS 

GDAAC 

MDSDS i 

• 3 VAX 64 WT" 
(14GB/DASD) 

• 2 SGI 4D/440 
(16 GB DASD) 

• IBM ES 9021/500 
(56 ch, 128 MB 



• IBM 3980 - (240 GB/DASD) 



• Convex 3240- (512 MB) 

• 2 Cygnet worm Jukeboxes 
with 4 SONY drives 

• 2 CYGNET WORM Juketox 
with 4 ATG 9001 drives (24 TB) 

• 1 Metnirn ACS (8.7TB) w/ 

• 3 STK 440 (4.8TB) 

• 1 Dataware WORM Jukebox 3 
34/850(1-2 TB) 


4 drives 

• B-Test Helical E-Systems 
Tower (8.2 TB) 

• 2 9-track Dec Upe drives 

• 2 8mm Exabyte tape drives 

• 2 4mm Winchester tape drives 

• CD-ROM prc-mastcring 

•1 I/O power channel 80MB/S 
•5 6250 9 track tape 
•4 Exabyte 8 mm tape drives 
•2 3480 tape cartridge 

• CD-Rom pre-mastering 

• 2 4mm DAT tape drives 

• 8 ESCON Channel 

• Ultranet 

• 40 Memorex 3480 

compatible tape drives 


605 













606 



Hardware Architecture Components 
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Goddard DAAC 








National Space Science Data Center 
NDADS Computer Cluster 


Cygnet 1802 Jukebox 
2 SONY WORM drtvM 
SI slots 
234 GBytes 



Cygnet 1803 Jukebox 
2 SONY WORM drlvos 
131 slots 
858 OBytos 


Stsndslont SONY optics! drivss 


Con troll sr dsvicss for 
psriphsrai equipment 
sccsssibis duster-wide 


HSC 


■ 




2 LAOO Taps Drivss 
•EXABYTE* 

S GByts capacity 


2 Winchester Taps drivss 
DAT 

S GByts capacity 


1 3 magnetic disk drivss 
20 GBytes storags 


4mm tops 
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Mass Storage Software Systems 


Software System 

NDADS 

GDAAC 

MDSDS 

Client/Scrver 
Operating System 

VAX VMS 

HR 

MVS, A IX 

Networks Supported 

DEClWt 

TCP/IP 

SPRINTnrt 

— TCF7IF 

~ TCP/lP 
Ethernet 
BITNET 
NSFNET 
InterNet 
InterLink 
UltraNet 

Database/Library Management 
System 

KST AC bii* S 1 Okt 
SYBASE 
INGRES 

Unitree 

ORACLE 

~ H5W 

Unitree 
Oracle (Opt) 

Physical Storage Device Driver 
Software 

JIW5 

SOAR 

Unitree 

Drivers 

Dataware 

STK 

Unitree Drivers 

User Interface 

NCDS 

ARMS* 

NSSDC ARCHIVE 

NCDS/PLDS 

EOSDIS/1MS 

USER CONTROL 


• AUTOMATED RETRIEVAL MAIL SYSTEM 


System Assessments 


NDADS 


GDAAC 


MDSDS 


Strengths: 


• Project customization 

• FTP Accessible 

• Distribution of archive 
media 

• Intelligent data access 
and optimization 

• Data compression 

• Metadata search/browse 


• Remote back-up 


• Project customization 

• FTP Accessible 

• Interoperable 

• Metadata search/browse 
NCDS/PCDS Experience 
Intelligent Data Mgmt. 
SoatialTemp Data Fusion 

a - 


• Metadata search/browse 

• IEEE Mass Storage 
Compliant 

• Open 


• FTP Accessible 
worldwide 

• Archival/retrieval by 
user request 

• Intelligent hierarchical 
storage migration 


• Remote back-up option 

• IEEE Mass Storage 
Compliant 


Weaknesses: 


• Non-portable systems 

• In-house customization 
of software and 
hardware 


• In-house customization 
of software 

• Embryonic HW/SW 
systems 


• Embryonic HW/SW 
systems 

• Costs 
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PERFORMANCE CHARACTERISTICS 


THESE ARE ALL PRELIMINARY ESTIMATES FOR THE NDADS M(DS)2 AT THIS POINT!! 

(DOES NOT INCLUDE PROJECT SPECIFICS) 


NDADS 

Read: 600 KB/sec 

Write: 300 KB/sec 

Read: 250 KB/sec 

Write: 107.52 KB/sec (average) 

250 KB/sec (max) 

11 seconds 
120 GBytes 
240 GBytes/year 

5 GB/day (average), 17 GB/day (max) 
330 filea/day, electronically 
24/day 


M(DS)2 


DASD: 80 MB/S Throughput 

Mass Storage: 18 MB/S Throughput 


SONY Optical Disk Drives: 
Actual Rates: 


Platter load speed: 

Data Storage: Current: 

Growth: 

Storage Input: 

Storage Output: 

Inquiries on the archive: 


CONCLUSION 

Mass storage systems allow scientists to perform research previously 
impossible because of logistic burdens and maintain pace with rapid 
data growth arising from increasing computational power and 
observational resolution 


Mass storage hardware systems technology evolving faster than 
software available to integrate into system. IEEE mass storage 
standards model changing faster than vendors can keep up with; still 
need standards. 


Community needs to acquire much more performance test data 
reliability, stability and data access speeds across small to large mass 
storage systems 


Not yet clear whether many small distributed client-servers are more 
effective than fewer large-scale client servers 


Mass storage management systems need to become more robust and 
more stable 
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